Skip to content

test(e2e): add per-feature verification suite and workflow - #255

Merged
jsugg merged 5 commits into
mainfrom
test/feature-verification-suite
Nov 11, 2026
Merged

jsugg merged 5 commits into
mainfrom
test/feature-verification-suite

Conversation

@jsugg

@jsugg jsugg commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Summary

Adds an end-to-end feature verification suite: one assertion-based test per StrongClaw feature, run against the real OpenClaw/ACPX versions that clawops bootstrap installs from platform/node-tools.

  • tests/feature_matrix.yaml: 102 features. Each lists its call sites (file:line), its test, its lanes (platforms are derived from the lanes), and a verification level (live, fake-model, stub). 9 features are unverifiable with a stated reason (live Telegram/WhatsApp, vendor coding-agent sessions, third-party vault backends, the TTY model wizard, the Langfuse stack, devflow run, and two doc-only procedures).
  • tests/suites/e2e/feature_verification/: the suite. It only runs inside a provisioned lane; elsewhere it is deselected, so make test and nested clawops baseline verify never touch it.
  • .github/workflows/feature-verification.yml: runs on workflow_dispatch and weekly. Lanes are linux-core (hypermemory, systemd activation), linux-lancedb, linux-profiles (no host services), and macos-launchd. A report job joins the lane results with the matrix and writes a per-feature PASS/FAIL/UNVERIFIABLE table to the step summary plus a feature-verification-report JSON artifact.
  • Model-dependent features use a local deterministic model server (GPT-provider- and Ollama-compatible API), declared as an explicit models.providers entry. Vendor ACP adapters are replaced by an ACP stub agent behind the real acpx. No real keys or secrets are used.
  • Reuses the fresh-host helpers: the job budget and deadlines (fix(ci): bound hosted docker waits by job-budget deadlines #238), image ensure, OrbStack runtime wait and the runtime-loss recovery (fix(ci): recover hosted macOS Docker runtime lost mid-scenario #252). There are three new scenarios; use_host_home puts Linux lanes in the runner HOME so systemd user units can be activated for real.
  • tests/suites/contracts/repo/test_feature_matrix_contract.py checks that:
    • every feature has a test and every suite test maps to exactly one feature;
    • every call site resolves;
    • every clawops command, profile, backend auth mode and compose stack is covered;
    • the workflow runs exactly the matrix lanes.

Baseline (openclaw 2026.3.13 / acpx 0.3.0)

A branch run recorded 73 PASS, 20 FAIL and 9 UNVERIFIABLE. The FAILs include several pre-existing product bugs, for example:

  • platform/systemd/openclaw-sidecars.service requires docker.service, which a user manager cannot resolve, so Linux service activation fails;
  • recovery/backends.py:49 passes a positional path to openclaw backup create, which takes --output;
  • memory-profile plugins.allow lists disable OpenClaw's bundled ollama provider;
  • the platform_verify.py:235 ss parser reads the peer column.

The post-merge baseline run and its full evidence are reported separately. This PR fixes none of the product bugs.

Add tests/feature_matrix.yaml with one entry per product feature (call
sites, test, lanes), a lane-scoped pytest suite that drives the real
clawops/openclaw/acpx install, a deterministic local model server, an
ACP stub agent for vendor adapters, and a feature-verification workflow
that reports PASS/FAIL/UNVERIFIABLE per feature.
Declare the fake model provider as an explicit models.providers entry
after setup, bring sidecars up through clawops when the service manager
did not, run hypermemory/context through varlock as the runtime does,
redact env-contract secrets from recorded results, and probe the fake
server without proxies.
Use OpenClaw's custom-local marker for the fake provider key, prove the
OTLP metrics pipeline end to end, read the Qdrant stamp from the stack's
own compose state dir, drive smart extraction past its message floor,
and split the default Ollama embedding lane into its own feature.
Reference the key through OpenClaw's provider env marker so it is never
stored or audited as plaintext, keep fresh-host's embedding default out
of lane commands, restore Qdrant when the reset feature fails, and let
bootstrap provision Node on lane hosts.
@jsugg
jsugg merged commit 1bcc633 into main Nov 11, 2026
38 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant